Skip to content

WIP Lots of improvements#898

Draft
snazy wants to merge 65 commits into
projectnessie:mainfrom
snazy:broad-review-fixes
Draft

WIP Lots of improvements#898
snazy wants to merge 65 commits into
projectnessie:mainfrom
snazy:broad-review-fixes

Conversation

@snazy

@snazy snazy commented Jul 16, 2026

Copy link
Copy Markdown
Member

No description provided.

@snazy
snazy force-pushed the broad-review-fixes branch 17 times, most recently from 5a3e357 to 4049104 Compare July 22, 2026 15:26
snazy added 13 commits July 22, 2026 20:08
Plan undecorated checked top-level variables as direct activation lookups instead of absolute attributes.

Select roots and decorated programs keep the attribute shape so qualifier planning, partial unknowns, and custom decorators retain their existing behavior.
Route unresolved varargs calls through a receiver-specific evaluator that avoids building the full argument array before receiver dispatch.

Also pass the complete receiver tail to dynamic receivers and cover unchecked receiver varargs calls with multiple arguments.
Includes some Java 17 specific improvements, not cosmetic fluff.
This change adds the necessary base work for the CEL-Spec bump.
The original Bazel-driven build in CEL-Java to run conformance-tests was leveraging the `simple_test` convenience binary to run tests.
That binary no longer exists for CEL-Spec 0.25.2 and was replaced with a Java client in the CEL-Spec version bump PR.

This change goes further and removes the Bazel build entirely and replaces the client/server split, which was needed for the `simple_test` tool, with JUnit test-factory.
Update string-to-bool conversion to accept the CEL-Spec spellings while rejecting unsupported mixed-case values.

Compare type values by CEL type name so well-known object type values match their built-in timestamp and duration type values.

Allow null assignment to wrapper fields and leave wrapper, message, timestamp, and duration fields unset during protobuf construction.

Support backtick-quoted field names in select expressions and message field initializers, then remove the now-passing conformance skips.
Handle CEL null values at the protobuf construction boundary for repeated and map fields whose value type is timestamp, duration, or a protobuf wrapper.

Convert map message values to the actual target protobuf message type instead of always using google.protobuf.Value.

Avoid hasField() for repeated wrapper fields when reading protobuf values, since repeated fields do not support presence checks.
The map Any null-retention conformance cases now pass after the protobuf map conversion changes.

Remove the proto2 and proto3 skip entries so the JUnit conformance suite covers those cases.
Protobuf message construction could let bad field conversions escape as Java runtime exceptions, which broke disabled-check conformance cases expecting evaluation errors.

Wrap field conversion and assignment failures from newValue() as CEL Err values, add provider coverage for invalid null assignments, and enable the remaining proto2/proto3 set_null conformance cases.
Include the CEL-Spec type_deduction suite in the JUnit conformance runner to cover checker result types for already-supported language features.

Keep explicit skips for the remaining unsupported checker cases around optionals, flexible type parameter generality, wrapper promotion, and legacy nullable generic candidates.
The dynamic float protobuf range-assignment cases now pass under the JUnit conformance runner.

Remove the stale proto2 and proto3 float range skip entries while keeping the int32 and uint32 range cases skipped because they still overflow into protobuf field values instead of returning range errors.
The legacy proto2 and proto3 standalone enum out-of-range assignment cases now pass under the JUnit conformance runner.

Remove the stale skip entries for the too-large and too-negative enum assignment cases.
snazy added 27 commits July 22, 2026 20:26
The updated CEL-Spec includes macros that expose both key/index and value variables during comprehension evaluation. CEL-Java only modeled one iteration variable, which prevented exists/all/existsOne and transform macros from covering list indexes and map values.

Add parser, checker, and interpreter support for the second comprehension variable, add transformList and transformMap macro expansion, preserve iter_var2 in AST pruning and exhaustive evaluation, and enable the macros2 conformance file with direct unit coverage.
Move CEL grammar generation into cel-core using CongoCC and remove the generated-antlr module, shadow jar dependency, and ANTLR runtime wiring.

The generated parser now builds internal AST nodes with typed CEL expression conversion hooks, keeping parser construction close to the grammar while preserving the existing parser API and source-location handling.

This simplifies dependency management for downstream consumers, removes the relocated ANTLR artifact from the build graph, and makes future grammar changes easier to review in the core module.
Add runtime optional values and wire optional constructors, receiver methods, lazy optMap/optFlatMap macros, and optional access operators.

Parse optional select, index, list, map, and message syntax with CongoCC, then preserve the protobuf optional-entry AST fields through checking and interpretation.

Enable optionals conformance coverage and remove the optional-dependent block-extension skips.
Struct-pruning was effectively doing nothing.
Convert MapT entries directly to protobuf MapEntry values instead of first materializing an intermediate Java map. Use the protobuf field descriptors for key and value conversion so scalar widths, unsigned values, enums, messages, and null handling retain their protobuf semantics.\n\nAdd regression coverage that rejects native-map materialization and exercises representative protobuf map value types.
Bind ordinary Jackson-discovered methods and fields to property-specific MethodHandles. Checked selectors can invoke the cached getter directly, while dynamic JacksonObjectT access performs one metadata lookup. Custom and inaccessible writers retain the BeanPropertyWriter fallback.

OpenJDK 21.0.11, Linux amd64, JMH 1.37, 2 threads, 3 forks:

Before:
```
  preResolvedPropertyRead: 0.009 +/- 0.001 us/op, approximately 0 B/op
  propertyRead:            0.031 +/- 0.003 us/op, 32.012 +/- 0.027 B/op
```
After:
```
  preResolvedPropertyRead: 0.009 +/- 0.001 -> 0.003 +/- 0.001 us/op (66.7% faster), approximately 0 B/op
  propertyRead:            0.031 +/- 0.003 -> 0.032 +/- 0.003 us/op (within error), about 32 B/op
```
Capture each Jackson-discovered BeanPropertyWriter in the cached FieldType. Checked selectors invoke Jackson 3s existing method-handle-backed accessor without a name lookup, while dynamic JacksonObjectT access performs one metadata lookup.

OpenJDK 21.0.11, Linux amd64, JMH 1.37, 2 threads, 3 forks:

Before:
```
  preResolvedPropertyRead: 0.006 +/- 0.001 us/op, approximately 0 B/op
  propertyRead:            0.035 +/- 0.005 us/op, 32.016 +/- 0.035 B/op
```
After:
```
  preResolvedPropertyRead: 0.006 +/- 0.001 -> 0.004 +/- 0.001 us/op (33.3% faster), approximately 0 B/op
  propertyRead:            0.035 +/- 0.005 -> 0.030 +/- 0.004 us/op (confidence intervals overlap), about 32 B/op
```
Bind validated MethodHandle getters for registered generated messages with singular bool, string, bytes, signed integer, float, and double fields. Dynamic messages, extensions, incompatible classes, and normalization-sensitive field categories retain descriptor access.

OpenJDK 21.0.11, Linux amd64, JMH 1.37, 2 threads, 3 forks:
Before:
```
  generatedScalarFieldGetter:      0.023 +/- 0.001 us/op, approximately 0 B/op
  benchmarkResolverFieldQualifier: 0.077 +/- 0.007 us/op, 24.001 +/- 0.001 B/op
```
After:
```
  generatedScalarFieldGetter:      0.023 +/- 0.001 -> 0.004 +/- 0.001 us/op (82.6% faster), approximately 0 B/op
  benchmarkResolverFieldQualifier: 0.077 +/- 0.007 -> 0.072 +/- 0.005 us/op (confidence intervals overlap), 24.001 B/op
```
Make TypeAdapter a bidirectional conversion boundary with exact-wrapper primitive input defaults, allocation-free built-in input overrides, strict primitive output helpers, and one compatibility fallback for custom Val implementations.

Mark `Val.convertToNative()` deprecated for removal while retaining its implementations as the compatibility surface. Preserve wrapper-token behavior, CEL int/uint range semantics, and `TypeAdapter`'s functional-interface contract.

Use each value container's or environment's `TypeAdapter` for outbound native conversions so conversion policy has a single internal seam. Remove `ObjectT`'s redundant `TypeAdapter` forwarding and keep the deprecated `Val` method only as the compatibility fallback in `TypeAdapterSupport`.

Thread the environment adapter through conformance result encoding and use direct values in type-checked string extension tests. This removes deprecated Val conversion calls from operational test paths while retaining the per-Val conversion tests as compatibility coverage.

Added focused native-to-Val, Val-to-native, and direct primitive getter benchmarks.

Benchmark results:
* Primitive boolean inbound: 3.451 → 0.438 ns/op
* Primitive long inbound: 5.988 → 2.215 ns/op
* Primitive double inbound: 4.120 → 2.215 ns/op
* Primitive outbound adapter methods: approximately 0.51 ns/op, effectively zero allocation
Benchmarked overload dispatch paths before this change:

Add focused JMH coverage for direct and evaluator calls at arities zero through four, receiver fallback with zero through four tail arguments, and bound generic trait success/fallback.
Add semantic characterization for checked and unchecked binding, receiver fallback, error and unknown propagation, cost, call shape, and state tracking.

Add fixed three/four/five-argument operation and evaluator while retaining the generic `FunctionOp` fallback.

Preserve generic error/unknown ordering, trait checks, receiver fallback, call metadata, cost, decorators, and no-such-overload formatting.

JMH baseline, using JDK 21.0.11, 3 forks, 5x200 ms warmup, 10x200 ms measurement, GC profiler:

| Path | Generic before | Fixed after | Allocation before | Allocation after |
| --- | ---: | ---: | ---: | ---: |
| evaluator, arity 3 | `9.628 ns/op` | `2.096 ns/op` | `32 B/op` | approximately `0 B/op` |
| `math.greatest`, arity 3 | `16.312 ns/op` | `3.073 ns/op` | `64 B/op` | approximately `0 B/op` |
| Strings `indexOf` with offset | `15.601 ns/op` | `10.419 ns/op` | `56 B/op` | `24 B/op` |
| evaluator, arity 4 | `12.524 ns/op` | `2.556 ns/op` | `32 B/op` | approximately `0 B/op` |
| `math.greatest`, arity 4 | `20.520 ns/op` | `3.994 ns/op` | `64 B/op` | approximately `0 B/op` |
| Strings `replace(..., n)` | `31.349 ns/op` | `24.956 ns/op` | `136 B/op` | `104 B/op` |
| evaluator, arity 5 | `14.927 ns/op` | `2.868 ns/op` | `40 B/op` | approximately `0 B/op` |
| `math.greatest`, arity 5 | `21.978 ns/op` | `4.450 ns/op` | `80 B/op` | approximately `0 B/op` |
Expand validated generated protobuf access to explicit presence, unsigned fields, numeric enums, wrappers, repeated fields, lazy maps, and PbObjectT-only ordinary messages. Reuse cached FieldType metadata in PbObjectT through allocation-free nested cache hits, invalidate entries when dynamic registrations are upgraded, and retain descriptor fallbacks for incompatible values and unsupported generated APIs.

Keep ordinary-message getters off checked deep chains: the experiment was latency-neutral and increased allocation, while restricting them to PbObjectT improved object message access. Add CEL-facing semantic coverage and a dedicated FieldAccessBench.

JMH 1.37, OpenJDK 21.0.11 Zulu, 3 forks, 5x200ms warmup, 10x200ms measurement, 1 thread, GC profiler (ns/op, B/op):

* checked terminal int32: 59.64 +/- 6.33, 88 B -> 38.96 +/- 4.01, 72 B
* PbObject generated int32: 30.81 +/- 2.79, 56 B -> 15.15 +/- 0.29, 40 B
* PbObject dynamic int32: 30.67 +/- 2.87, 56 B -> 14.56 +/- 0.41, 40 B
* raw presence present: 12.32 +/- 0.57, ~0 B -> 4.00 +/- 0.03, ~0 B
* raw enum: 36.41 +/- 3.78, 64 B -> 4.49 +/- 0.09, 16 B
* PbObject map selection: 149.71 +/- 9.22, 328 B -> 27.70 +/- 0.48, 72 B
* PbObject nested message: 52.16 +/- 3.08, 32 B -> 26.33 +/- 3.43, 32 B

Fallback remained neutral: DynamicMessage scalar was 17.83 +/- 0.08 -> 17.70 +/- 0.33 ns/op at ~0 B/op. Checked deep selection was 147.54 +/- 14.46 -> 159.82 +/- 14.24 ns/op at 120 B/op with overlapping intervals, which is why that path keeps descriptor message access.
Avoid internal CEL index allocation during list traversal and specialize exact primitive-array membership and equality. Snapshot native maps with eagerly adapted keys and lazily adapted values, use descriptor-aware native lookup for generated protobuf maps, and share representation-neutral map equality.

Canonicalize heterogeneous numeric hashes and preserve error/unknown and strict evaluation semantics for optimized membership and map operations. Add focused correctness coverage and CollectionOperationBench.

JMH 1.37, OpenJDK 21.0.11, 3 forks, 5x200ms warmup, 10x200ms measurement, 1 thread, avgt ns/op, GC profiler; 1024 entries:

* checked int[] membership: 6061.2 ns / 49216 B -> 164.1 ns / 96 B

* checked int[] equality: 6079.3 ns / 73824 B -> 77.8 ns / 40 B

* native-map membership: 34493.3 ns / 139161 B -> 26757.6 ns / 114633 B

* native-map equality: 100127.7 ns / 360195 B -> 94312.0 ns / 319131 B

* generated proto-map lookup: 10012.2 ns / 33112 B -> 96.4 ns / 248 B

* generated proto-map equality: 121813.5 ns / 465516 B -> 32406.7 ns / 115033 B

Verified with core tests, Spotless, Javadoc and JMH packaging; repository-wide tests; conformance; Jackson 2/3; standalone variants; and Quarkus smoke tests. GraalVM native-image was not available.
Add primitive size and integer-index trait methods with compatible defaults, direct built-in implementations, and checked BaseListT access. Use them in trusted traversal, conversion, capacity, formatting, pruning, math-list, and list-qualifier paths.

Traverse concrete map backing entries directly for equality and hashing, and compare compatible generated protobuf maps through native keys while retaining CEL value equality and generic fallbacks.

JMH 1.37, OpenJDK 21.0.11, 3 forks, 5x200ms warmup, 10x200ms measurement, 1 thread, avgt ns/op, GC profiler; 1024 entries:

* indexed Java-list scan: 6748.7 +/- 50.0 ns, 49128 B -> 6392.1 +/- 29.4 ns, 24576 B

* wrapped-map equality: 29198.1 +/- 587.8 ns, 82025 B -> 17668.6 +/- 870.8 ns, 40961 B

* native/wrapped-map equality: 37556.8 +/- 393.2 ns, 106553 B -> 23513.3 +/- 191.6 ns, 65489 B

* generated protobuf-map equality: 32088.8 +/- 463.8 ns, 115033 B -> 20648.5 +/- 586.1 ns, 49401 B

* native/native-map control: 28260.6 +/- 314.8 ns, 90017 B -> 28247.6 +/- 781.3 ns, 90017 B

A generic-list equality native-size experiment removed about 24 B/op but had inconsistent and sometimes slower latency, so that call site retains size().

Verified with repository-wide tests, Spotless, and Javadoc.
Reuse immutable CEL values for small signed and unsigned integers, -1/1 doubles and both IEEE-754 zero representations. Preserve the raw sign of negative zero and cover it with a regression test.

Reuse immutable string values when converting booleans and null to CEL strings.

Verified with focused scalar type tests, Spotless, and core Javadoc.
Prevent successful built-in activation lookup currently constructs a temporary `ResolvedValue`, which is then immediately unwrapped in `EvalIdent` or `AttributeFactory`.

This change eliminates the unnecessary object allocation.
Add checked-expression JMH baselines for raw evaluation, Program.eval(), native result conversion, and Java controls across core, protobuf, and Jackson inputs.

Centralize int, uint, and double comparison and equality behavior in a shared helper, with coverage for numeric boundaries, NaN, and signed zero.
@snazy
snazy force-pushed the broad-review-fixes branch from 4049104 to 60a56e8 Compare July 22, 2026 18:29
Introduce a conservative whole-expression evaluator for supported checked scalar expressions. Typed boolean, long, double, and string nodes keep successful intermediate values out of Val wrappers, while existing Err, UnknownT, null, activation, cost, and evaluation-order semantics remain intact. Unsupported syntax, options, decorators, adapters, field providers, or replaced overloads continue to use the standard interpreter.

Extend the eligible domain to checked map scalar selectors, characterized protobuf scalar fields, Jackson 3 string fields, constant indexes into top-level numeric and string lists, same-type string membership against top-level string lists and checked non-optional string-list literals, and checked non-optional integer, double, string, and boolean list literals fused with an immediate constant index or exact standard global/receiver size consumer. Explicit adapter and field-provider capabilities prevent custom implementations from being treated as semantically compatible, typed generated-protobuf getters avoid primitive boxing, and null-backed strings preserve exact existing StringT(null) behavior through localized compatibility transport.

The representation choice is measurement-driven. At depth 16, Object-returning nodes retained 384 B/op versus 408 B/op for the current mixed evaluator, while typed nodes were effectively allocation-free and reduced mixed long/double latency from 97.9/136.5 ns to 23.7/23.0 ns. The production Program boundary reduced depth-16 allocation from 464 B/op to 80 B/op (82.8 percent), with int latency improving from 118.9 to 71.3 ns and double from 102.4 to 46.2 ns.

Retained selector and list slices also remove their raw-path allocation: map selection improved from 30.9 ns / 24 B to 18.3 ns / approximately 0 B, generated protobuf int selection from 48.4 ns / 72 B to 18.2 ns / approximately 0 B, Jackson 3 string selection from 30.4 ns / 32 B to 20.3 ns / approximately 0 B, long-array comparison from 60.6 ns / 80 B to 12.1 ns / approximately 0 B, and String-array comparison from 54.8 ns / 64 B to 13.7 ns / approximately 0 B.

Raw String-array membership improved from 1114.8 ns / 48 B to 630.9 ns / approximately 0 B for an absent value at size 1024; the corresponding Program path improved from 1161.3 ns / 104 B to 759.7 ns / 40 B. Numeric membership was prototyped but rejected after absent size-1024 long-array latency regressed by 22 percent despite removing allocation.

Checked string-list literal membership completes literal construction before matching, preserves early list error/unknown termination and left needle precedence, and allocates compatibility storage only for exceptional successful runtime values. For absent computed size 1024, raw evaluation improved from 28703 ns / 8313 B to 9200 ns / approximately 0 B and Program evaluation from 27904 ns / 8369 B to 9152 ns / 56 B. The constant Program path improved from 7157 ns / 8336 B to 1974 ns / 40 B.

Checked scalar-list literal index fusion avoids materializing the aggregate while preserving complete construction order, delayed bounds errors, error/unknown precedence, selected and unselected runtime mismatches, null-backed strings, and double special values. For computed size 1024, integer raw evaluation improved from 20654.0 ns / 57473 B to 8271.2 ns / approximately 0 B, double from 16857.5 ns / 57449 B to 3184.5 ns / approximately 0 B, string from 32436.4 ns / 8297 B to 9090.5 ns / approximately 0 B, and boolean from 29059.3 ns / 8297 B to 7888.3 ns / approximately 0 B. The corresponding Program paths improved to 7382.8 ns / 56 B, 2605.6 ns / 64 B, 9307.1 ns / 72 B, and 8162.3 ns / 56 B respectively.

The exact standard global and receiver size consumers reuse the literal plan without producing an aggregate. Constant elements return their known cardinality directly; computed elements still evaluate sequentially, preserve the first error or unknown, and discard successful runtime type mismatches exactly like current list construction. For computed size 1024, integer raw evaluation improved from 28804.8 ns / 57473 B to 8174.7 ns / approximately 0 B, double from 16710.3 ns / 57473 B to 2731.2 ns / approximately 0 B, string from 27968.1 ns / 8321 B to 6829.6 ns / approximately 0 B, and boolean from 28240.7 ns / 8321 B to 6884.3 ns / approximately 0 B. Their Program paths improved to 8325.7 ns / 80 B, 2208.7 ns / 64 B, 7060.5 ns / 80 B, and 7004.1 ns / 80 B respectively.

Terminal scalar-list results remain on the interpreter because primitive arrays would change the observable ValListT/Object[] representation.

Add OptDisableNativeEval as an explicit interpreter control, use it for differential and benchmark oracles, and execute conformance in native-enabled and native-disabled modes. The dual run completed 4,908 cases with 4,838 passing and 70 skipped.

Validated with the core, Jackson 2, Jackson 3, and conformance test suites, Spotless, core Javadoc, and core/Jackson 3 JMH jar construction.

Consolidate strict scalar dispatch behind package-private native-overload descriptors keyed by the checked overload ID and exact resolved implementation identity. Standard descriptors reuse the trusted standard-overload identity set, while custom registration is lazy and unregistered or incorrectly bound implementations fall back to the interpreter. Preserve the original switch-shaped arithmetic nodes after generic operation lambdas regressed deep latency; paired HEAD controls retain effectively zero primitive allocation, 24 B/op final adaptation, neutral steady-state latency, and identical supported-plan construction allocation. Keep this extension seam internal until a real external library consumer justifies a public API; macros remain AST expansion.

Evaluate the exact canonical checked single-variable exists expansion over compatible top-level scalar lists with a typed evaluation-local loop binding. Match the checked comprehension shape independently of macro provenance, and preserve current fold scoping, short-circuit error and unknown suppression, cost, fallback, and concurrent plan reuse. Nested, two-variable, non-canonical, or unsupported predicates remain on the interpreter.

Direct primitive and raw-string array traversal removes per-element Val and activation allocation. In three-fork JDK 21 size-1024 late-match Program.eval measurements, long[] improved from 76,932 ns / 49,027 B to 7,405 ns / 104 B, while String[] improved from 63,338 ns / 32,906 B to 8,450 ns / 104 B.

Generalize canonical quantifier planning to the exact checked single-variable all expansion over the same top-level scalar-list domain. Share one typed traversal and 48-byte loop binding with exists while preserving empty-range truth, first-false short-circuiting, later-false suppression of earlier errors and unknowns, fold scope, cost, fallback, and concurrent reuse. Primitive boolean arrays retain the current adapter error rather than bypassing compatibility.

In three-fork JDK 21 size-1024 late-failure Program.eval measurements, long[] improved from 67,814 ns / 49,026 B to 7,664 ns / 104 B, while String[] improved from 60,284 ns / 32,906 B to 9,233 ns / 104 B. A first/last/all-true matrix confirmed early termination and fixed allocation; the generalized exists control retained 48 B raw and 104 B public allocation and remained within the 10-percent latency gate.
@snazy
snazy force-pushed the broad-review-fixes branch from 60a56e8 to 57f38e8 Compare July 22, 2026 21:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant